Scene-level Tracking and Reconstruction without Object Priors

We present the first real-time system capable of tracking and reconstructing, individually, every visible object in a given scene, without any form of prior on the rigidness of the objects, texture existence, or object category. In contrast with previous methods such as Co-Fusion and MaskFusion that first segment the scene into individual objects and then process each object independently, the proposed method dynamically segments the non-rigid scene as part of the tracking and reconstruction process. When new measurements indicate topology change, reconstructed models are updated in real-time to reflect that change. Our proposed system can provide the live geometry and deformation of all visible objects in a novel scene in real-time, which makes it possible to be integrated seamlessly into numerous existing robotics applications that rely on object models for grasping and manipulation. The capabilities of the proposed system are demonstrated in challenging scenes that contain multiple rigid and non-rigid objects.Comment: Accepted by IROS202

Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs

We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as ``pick up a cup on a kitchen table" or ``navigate to a sofa on which someone is sitting". In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments.Comment: The code and dataset used for evaluation can be found at https://github.com/changhaonan/OVSG}{https://github.com/changhaonan/OVSG. This paper has been accepted by CoRL202

Shift Current Photovoltaics based on A Noncentrosymmetric Phase in in‐plane Ferroelectric SnS

原子層強誘電材料のバルク光起電力発電を実証 --ナノ発電実現へ新たな道を開拓--. 京都大学プレスリリース. 2023-06-09.The shift-current photovoltaics of group-IV monochalcogenides has been predicted to be comparable to those of state-of-the-art Si-based solar cells. However, its exploration has been prevented from the centrosymmetric layer stacking in the thermodynamically stable bulk crystal. Herein, the non-centrosymmetric layer stacking of tin sulfide (SnS) is stabilized in the bottom regions of SnS crystals grown on a van der Waals substrate by physical vapor deposition and the shift current of SnS, by combining the polarization angle dependence and circular photogalvanic effect, is demonstrated. Furthermore, 180° ferroelectric domains in SnS are verified through both piezoresponse force microscopy and shift-current mapping techniques. Based on these results, an atomic model of the ferroelectric domain boundary is proposed. The direct observation of shift current and ferroelectric domains reported herein paves a new path for future studies on shift-current photovoltaics

Model-enhanced Vector Index

Embedding-based retrieval methods construct vector indices to search for document representations that are most similar to the query representations. They are widely used in document retrieval due to low latency and decent recall performance. Recent research indicates that deep retrieval solutions offer better model quality, but are hindered by unacceptable serving latency and the inability to support document updates. In this paper, we aim to enhance the vector index with end-to-end deep generative models, leveraging the differentiable advantages of deep retrieval models while maintaining desirable serving efficiency. We propose Model-enhanced Vector Index (MEVI), a differentiable model-enhanced index empowered by a twin-tower representation model. MEVI leverages a Residual Quantization (RQ) codebook to bridge the sequence-to-sequence deep retrieval and embedding-based models. To substantially reduce the inference time, instead of decoding the unique document ids in long sequential steps, we first generate some semantic virtual cluster ids of candidate documents in a small number of steps, and then leverage the well-adapted embedding vectors to further perform a fine-grained search for the relevant documents in the candidate virtual clusters. We empirically show that our model achieves better performance on the commonly used academic benchmarks MSMARCO Passage and Natural Questions, with comparable serving latency to dense retrieval solutions